Identifying collection images with special events

ABSTRACT

A method for associating event times or time periods with digital images in a collection for determining if a digital image is of interest, includes storing a collection of digital images each having an associated capture time; comparing the associated capture time in the collection with a special event time to determine if a digital image in the collection is of interest, wherein the comparing step includes calculation of a special event time associated with a special event based on the calendar time associated with the special event and using such information to perform the comparison step; and associating digital images of interest with the special event.

CROSS REFERENCE TO RELATED APPLICATIONS

Reference is made to commonly assigned U.S. patent application Ser. No.11/116,279, filed Apr. 28, 2005, entitled “Using Time in RecognizingPersons in Images” by Gallagher et al and U.S. patent application Ser.No. ______ filed Jun. 2, 2005, entitled “Using Photographer Identity toClassify Images” by Gallagher et al, the disclosures of which areincorporated herein.

FIELD OF THE INVENTION

The present invention related to an improved way to identify digitalimages from a collection, making use of special events.

BACKGROUND OF THE INVENTION

With the advent of digital photography, consumers are amassing largemedia collections of digital images and videos. For purposes of thisdescription, the term “images” will be understood to include both stillimages and “videos” which are a collection of image frames, often havingassociated audio stream. Therefore, an image collection can containimages or videos or both. The average number of images captures withdigital cameras per photographer is still increasing each year. As aconsequence, the organization and retrieval of images and videos isalready a problem for the typical consumer. Currently, the length oftime spanned by a typical consumer's digital image collection is only afew years. The organization and retrieval problem will continue to growas the length of time spanned by the average digital image and videocollection increases.

The automatic organization of a media collection, either as an end initself or for use in other applications—has been the subject of recentresearch. Relatively sophisticated image content analysis techniqueshave been used for image indexing and organization. For image indexingand retrieval applications, simple text analysis techniques have alsobeen used on text or spoken annotations associated with individualimages or videos. The recent research has involved a number oftechniques and tools for automatic albuming of images and videos.

Date and time information from the camera has been used to perform eventsegmentation, as for example described in U.S. Pat. No. 6,606,411 byLoui and Pavie. An event consists of a set of images of videos relatedto a common event, for example “trip to the beach.”

U.S. Pat. No. 6,810,146 by Loui and Stent, described extracting certaintypes of information from spoken annotations, or the transcriptions ofspoken annotations, associated with photographs, and then using theresults to perform event segmentation, identification, andsummarization.

Certain applications allow for viewing images on a timeline. In essence,the images can be viewed or sorted into consecutive order based on theimage capture time. For example, the application Picasa 2, distributedby Google, has a timeline view where groups or sets of images and videosare shown to the user and each image set has an associated time (e.g.“April 2005”). In addition, the Adobe application Album 2.0 has acalendar view where calendar pages are shown and small versions ofimages captured on a specific calendar date are shown on that date. Ineach case, the software groups images related by capture time into sets.However, the sets of images are not labeled with meaningful names otherthan the capture date or date range. Thus, the only calendar informationused by these applications is the day, month, and year. They do not useany occasion (e.g. Thanksgiving) or appointment (e.g. vacation trip toFlorida) information to label the images with meaningful names.

Furthermore, U.S. Pat. No. 6,108,640 describes a method for determiningperiodic occasions such as holidays. However, there is no description ofassigning meaningful labels to images or sets of images.

In UK Pat. Application GB2403304A, Rowe describes a method of labelingimages with labels based on the image capture dates corresponding tonational events for later use in text-based search and retrieval ofimages. This method cannot provide for the fact that for many people,many national holidays are not observed. For example, few peopleactually celebrate Groundhog Day. Subsequent searches by a user forimages of “groundhogs” would return images captured on February 2.Furthermore, since may consumer images are taken on occasions that arenot associated with national holidays (such as the birthdays of familymembers, or vacation trips), this method cannot provide useful labelsfor most consumer photos.

In U.S. Patent Application Publication US20040201740A1, Nakamura andGibson describe a method of placing images into storage locations basedon calendar information. Their method does not provide for automaticannotation of images.

In U.S. Pat. application Publication US20050044066 A1, Hooper and Maodescribe a calendar-based image asset organization method. Their methodallows people to indicate via a graphical user interface a date range ofinterest. Images captured during that date range are then retrieved forthe user. The method does not provide for automatic annotation ofimages.

SUMMARY OF THE INVENTION

An object of the present invention to provide an improved way ofidentifying digital images of interest from a collection of digitalimages.

This object is achieved by storing a collection of digital images orvideos or both each having an associated capture time; comparing theassociated capture time in the collection with a special event time todetermine if a digital image or video in the collection is of interest,wherein the comparing step includes calculation of a special event timeassociated with a special event based on the calendar time associatedwith the special event and using such information to perform thecomparison step; and associating digital images and videos of interestwith the special event.

Another object of the present invention is to provide an improved way tolabeling digital images captured by a digital capture device.

This object is achieved by transferring and storing calendar informationin a digital camera device, the calendar information including occasionor appointment information, capturing and storing a digital image in thedigital camera device, determining a capture time for the captureddigital image, automatically comparing the capture time with thecalendar information to determine a special event label, and storing thespecial event label in association with the captured digital image.

It has been found that digital images of interest can be effectivelysearched in a collection of digital images using special eventinformation that has associated event time or time period information.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned and other features and objects of this invention andthe manner of attaining them will become more apparent and the inventionitself will be better understood by reference to the followingdescription of an embodiment of the invention taken in conjunction withthe accompanying figures wherein:

FIG. 1 is a schematic diagram of computer system that can implement thepresent invention;

FIG. 2 is a flow chart of an embodiment of the present invention;

FIG. 3 is a more detailed flow chart of an embodiment of the presentinvention;

FIG. 4 is graph representation of a calendar time (cross hatch region)and a special event time (line) associated with the special eventChristmas;

FIG. 5 is a flow chart of an embodiment of the present invention;

FIG. 6 is a flow chart of a further embodiment of the present invention;

FIG. 7 is a flow chart of yet another embodiment of the presentinvention;

FIG. 8 is a flow chart of another embodiment of the present invention;

FIG. 9 is a flow chart of a still further embodiment of the presentinvention;

FIG. 10 is a block diagram of a camera phone based imaging system thatcan implement the present invention;

FIG. 11 is a flow chart of yet another embodiment of the presentinvention;

FIG. 12 is a flow chart of a still further embodiment of the presentinvention; and

FIG. 13 is a flow chart of an additional embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, some embodiments of the present inventionwill be described as software programs. Those skilled in the art willreadily recognize that the equivalent of such software can also beconstructed in hardware. Because image manipulation algorithms andsystems are well known, the present description will be directed inparticular to algorithms and systems forming part of, or cooperatingmore directly with, the method in accordance with the present invention.Other aspects of such algorithms and systems, and hardware or softwarefor producing and otherwise processing the image signals involvedtherewith, not specifically shown or described herein can be selectedfrom such systems, algorithms, components, and elements known in theart. Given the description as set forth in the following specification,all software implementation thereof is conventional and within theordinary skill in such arts.

The present invention can be implemented in computer hardware andcomputerized equipment. For example, the method can be performed in adigital camera (as will be described later in reference to FIG. 10), adigital printer, on an internet server, on a kiosk, and on a personalcomputer. Referring to FIG. 1, there is illustrated a computer systemfor implementing the present invention. Although the computer system isshown for the purpose of illustrating a preferred embodiment, thepresent invention is not limited to the computer system shown, but canbe used on any electronic processing system such as found in digitalcameras, home computers, kiosks, retail or wholesale photofinishing, orany other system for the processing of digital images. The computersystem includes a microprocessor-based unit 20 (also referred to hereinas a digital image processor) for receiving and processing softwareprograms and for performing other processing functions. The digitalimage processor 20 processes images from image capture devices 10 suchas cameras, scanners, or computer image generation software. The digitalimage processor 20 can be used to process digital images to makeadjustments for overall brightness, tone scale, image structure, etc. ofdigital images in a manner such that a pleasing looking image isproduced by an image output device. The digital image processor 20interfaces with the general control computer 40 (also a microprocessorbased unit) for exchanging data and commands. The general controlcomputer 40 and the digital image processor 20 can be two differentmicroprocessors, or the functions of each can be performs by a singlephysical microprocessor. The digital image processor 20 often outputs animage to an image output device 30 for example a printer for displayingthe image. A display device 50 is electrically connected to the digitalimage processor 20 for displaying user-related information associatedwith the software, e.g., by means of a graphical user interface. Akeyboard 60 is also connected to the microprocessor based unit 20 viathe general control computer 40 for permitting a user to inputinformation to the software. As an alternative to using the keyboard 60for input, a mouse can be used for moving a selector on the displaydevice 50 and for selecting an item on which the selector overlays, asis well known in the art. Digital images and other data can also bestored on an offline memory device 70 such as an external hard drive,flash media, a drive that writes to CD-ROM or DVD media, or the like.

A compact disk-read only memory (CD-ROM) which typically includessoftware programs, is inserted into the general control computer 40 forproviding a means of inputting the software programs and otherinformation to the general control computer 40 and the digital imageprocessor 20. In addition, a floppy disk can also include a softwareprogram, and is inserted into the general control computer 40 forinputting the software program. Still further, the general controlcomputer 40 can be programmed, as is well known in the art, for storingthe software program internally. The general control computer 40 canhave a network connection, such as a telephone line or wirelessconnection, to an external network such as a local area network or theInternet (370 in FIG. 10).

Images can also be displayed on the display device 50 via a Flash EPROMmemory card, such as the well-known PC Card, Compact Flash, SD,MemoryStick cards.

The image output device 30 provides a final image. The image outputdevice 30 can be a printer or other output device that provides a paperor other hard copy final image. The image output device 30 can also bean output device that provides the final image as a digital file. Theimage output device 30 can also include combinations of output, such asa printed image and a digital file on a memory unit, such as a CD orDVD.

A digital image includes one or more digital image channels or colorcomponents. Each digital image channel is a two-dimensional array ofpixels. Each pixel value relates to the amount of light received by theimaging capture device corresponding to the physical region of pixel.For color imaging applications, a digital image will often consist ofred, green, and blue digital image channels. Motion imaging applicationscan be thought of as a sequence of digital images. Those skilled in theart will recognize that the present invention can be applied to, but isnot limited to, a digital image channel for any of the herein-mentionedapplications. Although a digital image channel is described as a twodimensional array of pixel values arranged by rows and columns, thoseskilled in the art will recognize that the present invention can beapplied to non rectilinear arrays with equal effect. Those skilled inthe art will also recognize that for digital image processing stepsdescribed herein below as replacing original pixel values with processedpixel values is functionally equivalent to describing the sameprocessing steps as generating a new digital image with the processedpixel values while retaining the original pixel values.

It should also be noted that the present invention can be implemented ina combination of software or hardware and is not limited to devices thatare physically connected or located within the same physical location.One or more of the devices illustrated in FIG. 1 can be located remotelyand can be connected via a network. One or more of the devices can beconnected wirelessly, such as by a radio-frequency link, either directlyor via a network.

The present invention can be employed in a variety of user contexts andenvironments. Exemplary contexts and environments include, withoutlimitation, wholesale digital photofinishing (which involves exemplaryprocess steps or stages such as film in, digital processing, printsout), retail digital photofinishing (film in, digital processing, printsout), home printing (home scanned film or digital images, digitalprocessing, prints out), desktop software (software that appliesalgorithms to digital prints to make them better—or even just to changethem), digital fulfillment (digital images in—from media or over theweb, digital processing, with images out—in digital form on media,digital form over the web, or printed on hard-copy prints), kiosks(digital or scanned input, digital processing, digital or hard copyoutput), mobile devices (e.g., PDA or cell phone that can be used as aprocessing unit, a display unit, or a unit to give processinginstructions), and as a service offered via the World Wide Web.

In each case, the invention can stand alone or can be a component of alarger system solution. Furthermore, human interfaces, e.g., thescanning or input, the digital processing, the display to a user (ifneeded), the input of user requests or processing instructions (ifneeded), the output, can each be on the same or different devices andphysical locations, and communication between the devices and locationscan be via public or private network connections, or media basedcommunication. Where consistent with the foregoing disclosure of thepresent invention, the method of the invention can be fully automatic,can have user input (be fully or partially manual), can have user oroperator review to accept/reject the result, or can be assisted bymetadata (metadata that can be user supplied, supplied by a measuringdevice (e.g. in a camera), or determined by an algorithm). Moreover, thealgorithm(s) can interface with a variety of workflow user interfaceschemes.

The invention is inclusive of combinations of the embodiments describedherein. References to “a particular embodiment” and the like refer tofeatures that are present in at least one embodiment of the invention.Separate references to “an embodiment” or “particular embodiments” orthe like do not necessarily refer to the same embodiment or embodiments;however, such embodiments are not mutually exclusive, unless soindicated or as are readily apparent to one of skill in the art.

An embodiment of the invention is illustrated in FIG. 2. A digital imageand video 104 is input to a calendar label annotator 114. It should beunderstood that herein when referring to digital images, digital videos,being a collection of digital images, are also included. Special eventsfrom a personal calendar 146 are used by the calendar label annotator114 to annotate the digital images and videos in the collection 104. Thecollection 104 can contain exclusively images, or exclusive videos, orsome of each. Resulting annotations 118 (equivalently called labels) arestored in a database 120. Each annotation 118 can be stored with theimage (e.g. in the file header) or in a database 120 either located withthe digital image or video or remotely (e.g. on a computer server). Thedatabase 120 can be in any form. In addition, the database 120 can bedistributed across many files or few files. Consequently, the database120 can be queried 122 to find those images and videos 104 containingthe query special event. For example, a query 122 for images of “DaytonaBeach” returns query results 124 containing the set of digital images orvideos 104 with a related annotation of “vacation to Daytona Beach.”Then query results 124 can be returned. The query results 124 are theset of digital images and videos 104 associated with the query specialevent.

The digital image and videos from the collection 104 are passed to acapture time extractor 108. The capture time extractor 108 determinesthe time each digital image or video 104 was captured, and outputs imagecapture times 112. The image capture time 112 of the digital image orvideo 104 is determined by one of several methods by the capture timeextractor 108. Typically the capture time is embedded in the file headerof the digital image or video 104. For example, the EXIF image format(described at www.exif.org) allows the image capture device 10 to storeinformation associated with the digital image or video 104 in the fileheader. The “Date\Time” entry is associated with the date and time theimage was captured. In some cases, the digital image or video 104results from scanning film or prints and the image capture time 112 isdetermined by detection of the date exposed into the image area (as isoften done at capture time), usually in the lower right corner of theimage. The date a photograph is printed is often printed on the back ofthe print. Alternatively, some film systems (such as APS) contain amagnetic layer in the film for storing information such as the capturedate. The capture time extractor 108 uses the most appropriate methodfor extracting the image capture time 112 of the image. Preferably, thesource of the digital images and videos 104 is a digital camera, and thecapture time extractor 108 extracts the capture time from the fileinformation.

Note that the image capture time 112 can be a precise minute in time,e.g. Dec. 9, 2000 at 10:00 AM. Or the image capture time 112 can be lessprecise, e.g. 2000 or December 2000. The image capture time 112 can bein the form of a probability distribution function e.g. Dec. 9, 2000±2days with 95% confidence. The image capture time 112 is input to thecalendar label annotator 114.

The personal calendar 146 is preferably a software application runningon a computer that is useful for recording special events. For example,the special events can be input to the personal calendar 146 by the useror by others or other applications. The special events recorded in thepersonal calendar 146 are of three types; occasions, appointments, andjournal entries. Essentially, the special events are labels for blocksof time that are personalized to the user (or users) of the calendar.For example, Outlook by Microsoft and Notes by Lotus are both examplesof calendar applications. The personal calendar 146 can also be anapplication that operates on a handheld device, such as a personaldigital assistant (PDA) or a cellular telephone camera, as will bedescribed later in reference to FIG. 11. The personal calendar 146 canalso be an application that runs on a server and is accessed by the uservia the Internet.

The first type of the three special events recorded in a personalcalendar 146 is an occasion 142. The occasion 142 is a periodicallyoccurring special event, celebration, religious holiday (fixed ormoveable), national holiday, festival, or the like. In general,occasions 142 can be computed in advance for any time period becausemathematical formulas are used to determine the date of the occasion 142in each subsequent year, as described by U.S. Pat. No. 6,108,640. Ingeneral, an occasion 142 is a name placed on an entire day. For example,every year Christmas is December 25. The occasions 142 celebrated by,observed by, or relevant to a particular person depend on a number offactors, including geographical, religious, and political factors. Theuser can indicate to the personal calendar 146 which occasions 142 arerelevant for him or her via user input 150. The user can select relevantholidays from a list, or select between sets of holidays (i.e. Canadianholidays vs. American holidays), or indicate the geographic, religious,and political factors and have the computer guess at the set of relevantoccasions to the user, which can then be refined through further userinput 150. While holidays such as Easter, Christmas, Hanukkah and thelike are widely observed; most people also celebrate personal occasions142 such as birthday anniversaries, wedding anniversaries, etc. Thesebirthday and wedding anniversaries can be for the user's family andclose friends. The user indicates these personal occasions 142 via userinput 150 to the personal calendar 146. The user can indicate specialevents by user inspection (i.e. selecting from lists of occasionspresented to the user via the display device 50 as shown in FIG. 1) orautomatically or both.

A second type of special event is an appointment 140. Appointment 140 isa special event describing something planned to take place in the futurethat is relevant for the user or users of the personal calendar 146. Ingeneral, appointments 140 are not names placed on an entire day (as isthe case with occasions). For example, an appointment 140 can be “doctorappointment at 2” or “Matthew's cub scout meeting at 6 PM” or also “meetSarah at the zoo at 4 PM”. The appointment 140 can span several days orweeks, such as “Camping vacation Aug. 9-14, 2004”.

The third type of special event is a journal entry 144. The journalentry 144 is an entry to the personal calendar 146 describing eventsthat have occurred in the past (as of the time of entry of the journalentry 144.) For example, the special event on Nov. 24, 2004 “Jonah firstcrawled today” is a journal entry because it describes events thatalready took place at the time of entry. Some journal entries 144 arequite lengthy, such as a diary where a user enters a few paragraphsdescribing the events and reflections of the day. These diary entriesare journal entries 144 because they are associated with a particularcalendar time, and are related to events associated with that calendartime. A modern form of the diary is a blog (or weblog) that allows usersto keep an on-line diary on the Internet (e.g. www.blogger.com.)

Each special event has an associated calendar time. As previouslystated, the calendar time associated with an occasion 142 is a singleday. The calendar time associated with an appointment 140 can be aprecise moment in time or a range of time. The range of time can be anylength; seconds, minutes, days or weeks long for example. In fact, thecalendar time associated with an appointment 140 can have only astarting time indicated (e.g. “doctor appointment at 2”). In this case,the appointment 140 ending time is estimated (e.g. to be 2 hours long).The journal entry's associated calendar time can be a precise moment intime or a range of time. The calendar time associated with a specialevent can be in the form of a probability distribution function e.g. anormal distribution centered on Dec. 9, 2000 at 10:00 AM with standarddeviation of 2 days.

The calendar label annotator 114 compares the image capture time 118with the special event times from the personal calendar 146. Specialevent times will be described further herein below. The calendar labelannotator 114 can reside on a camera, a personal computer, a server, acellular telephone, a PDA, or the like. The calendar label annotator 114determines digital images and videos of interest from the collection 104by comparing the associated image capture times 112 with the specialevent times associated with special events. Each digital image or videoof interest is then associated with its corresponding special event,preferably by producing an annotation 118 stored in a database 120indicating that the digital image or video 104 is associated with thespecial event.

FIG. 3 shows a more detailed view of the calendar label annotator 114.The event processor 152 processes the special events from the personalcalendar 146. The special events from the personal calendar 146 eachhave an associated calendar time, as previously described. In addition,the special events can have other information from user input 150. Forexample, each special event can have labeled whether it is an “imagingevent” or not by the user. An imaging event is a special event thatmight correspond to digital images or videos 104 in the user's digitalimage and video collection 104. For example, the appointment “Vacationto the Beach” from Aug. 12-19, 2005 probably is an imaging event, whilethe appointment “dentist” at 4:00 Aug. 2, 2005 probably is not.

The event processor 152 determines a special event time for each specialevent. The special event time related to the calendar time associatedwith the special event, but generally has a broader time span to accountfor the fact that images related to a special event need not occurduring the calendar time associated with the special event. The specialevent time is necessary because it is common for images and videosrelated to a special event to occur slightly before or after thecalendar time associated with the special event. For example, a child'sbirthday occasion can be June 26, but the birthday party can be held onJune 23. As another example, it is common that many images relate toChristmas even though captured before December 25. Images of setting upa Christmas tree, singing Christmas carols, or visiting St. Nicolas atthe mall can occur days or weeks prior to the calendar time (Dec. 25)associated with the special event occasion of Christmas. The eventprocessor 152 sets a special event time for each special event usinguser input 150 or a rule-based system. As with the image capture time112 and the calendar time associated with special events, the specialevent time can be a period of time or represented as a probabilitydistribution function. For example the user can indicate that thespecial event time associated with Christmas is the period of timebeginning on Dec. 13 and ending on Dec. 26. The user can also indicatethat the special event time for all birthdays is the period of time from3 days prior to 3 days following the calendar time of the birthday. Forexample, FIG. 4 shows an illustrative plot of the calendar time and thespecial event time associated with Christmas 2004. The calendar time isshown as a hatched area and the special event time is shown with a line.If the user does not indicate special event times, then the followingdefault rules are used to determine the special event time from thecalendar time for a special event: Appointments: a = b = 0.1 (q-w)Journal Entries: a = b = 0 Occasions: Birthdays/anniversaries a = b = 3days Christmas a = 20 days, b = 1 day Other occasions a = b = 1 daysThe special event time is computed as a time period beginning at timew−a, and ending at time q+b, where w is the beginning of the calendartime associated with the special event (or 5% of the calendar time whenthe calendar time is a probability distribution function p(x)), and q isthe end of the calendar time associated with the special event (or 95%of the calendar time when the calendar time is a probabilitydistribution function p(x)).

By inspection of the equations, it can be seen that the special eventtime associated with a special event is calculated from the calendartime associated with the special event.

The collection of special event times, whether automatically generatedor manually entered, is a list of special event times.

Referring again to FIG. 3, the event processor 152 also determines thelikelihood that special events from the personal calendar 146 areimaging events as previously mentioned. This determination can be basedon user input 150 where the user indicates whether an event is animaging event or not. Furthermore, in the absence of user input 150, theevent processor 152 can automatically determine the likelihood that aspecial event is an imaging event. This is accomplished by collectingthe images collections and personal calendars from a large number ofpeople and manually determining the relationships between images andvideos in the collection 104 and special events in the calendar. Thelikelihood that special events are imaging events can therefore belearned in such fashion using well-known machine learning techniques.For example, it is extremely rare that images in a collection 104correspond to appointments for medical treatment such as doctor anddentist visits. Therefore, the event processor 152 assigns a likelihoodto each special event indicating the likelihood that the special eventis an imaging event. The likelihood can be a probability or can be abinary flag. This flag is used to improve the accuracy of the annotationprovided by the present invention. For example, images that happen bycoincidence to have an image capture time 112 also corresponding to aspecial event time would normally be incorrectly annotated by thespecial event name. For example, a doctor appointment ended earlier thanexpected, then the person walked to a park and photographed ducks. Theimages of ducks would erroneously be associated with the annotation“Doctor Appointment” were it not that the event processor 152 determinedthat the doctor appointment is not an imaging event.)

The event processor 152 outputs a special event list 154. The specialevent list 154 is a list of special events including the name of thespecial event, the associated calendar time, and the associated specialevent time. The special event list 154 also indicates the likelihoodthat each special event is an imaging event as previously mentioned.Additional information such as the people present at the special event,the location of the special event (e.g. my house, the park, Erie, Pa.,etc.) can also be included for special events in the list.

The time comparer 156 compares the associated capture times for thedigital images and videos in the collection 104 with a special eventtime or time period to determine digital images and videos of interest158. Digital images and videos of interest 158 are those images andvideos from the digital image and video collection having an imagecapture time 112 coincident with the special event time associated witha special event. For example, an image captured on Dec. 24, 2004 is animage of interest 158 associated with the special event occasionChristmas when the special event time associated with the occasionChristmas is as shown in FIG. 4. The image would then receive theannotation 118 “Christmas” that would be stored in association with theimage in a database 120. Then, when a user places a query 122 for imagesof “Christmas”, the database 120 is searched by well-known techniquesand the aforementioned image with image capture time 112 of Dec. 24,2004 will be among the query results 124. The query results can then beshown on the display device 50 such as shown in FIG. 1.

An annotation 118 associated with a digital image or video of interest158 is special event identifier information and can contain multiplefields, such as: the name of the special event, the associated calendartime, and the associated special event time, the number of andidentities of people present at the special event (e.g. guest list for abirthday party), the location of the special event (e.g. my house, thepark, Erie, Pa., etc.). The annotation 118 is information related to thespecial event. Note that the annotations 118 made automatically bycomparing the image capture time to the special event time can be markedin the database 120 as being automatically generated. This allows a userto provide a query 122 and search the database 120 for matchesconsidering just manually entered annotations 118, just automaticallygenerated annotations 118, or combinations thereof as desired by theuser.

Not all of the images and videos in the collection 104 will have capturetimes 112 that correspond to special event times associated with eventsin the special event list 154. In other words, not all of the digitalimages and videos from the collection 104 will be images and videos ofinterest 158. These images can be shown to the user on the displaydevice 50 such as shown in FIG. 1 for manual labeling.

The time comparer 156 can output a confidence score associated with eachimage of interest 158 that depends on both the image capture time 112 ofthe image of interest and the special event time of the special event.Preferably, the confidence score is the value of the p(x) evaluatedwhere x is the image capture time 112 (or mean image capture time whenthe image capture time is a probability distribution function).

An alternative embodiment is illustrated in FIG. 5. In this embodiment,the digital image and video collection 104 is input to an eventclusterer 162 for clustering the digital images and videos 104 intoevent clusters 164 which are mutually exclusive smaller sets of digitalimages and videos 104 according to image content and image capture times112. Each set of digital images and videos 104 corresponds to achronologically related segmentation of the digital image and videocollection 104. In other words, each event cluster 164 contains allimages from the digital image and video collection 104 captured afterthe earliest image or video in the event cluster and before the latestimage in the event cluster 164. Commonly assigned U.S. Pat. No.6,606,411 to Loui and Pavie, incorporated herein by reference, describesthe production of event clusters 164 from a set of digital images andvideos 104 in detail.

The capture time extractor 108 then inputs each event cluster 164 anddetermined the image capture times 112 associated with each eventcluster 164. Preferably, the image capture time 112 for a particularevent cluster 164 is a time period or time range spanning the timebetween the earliest and the latest image in the event cluster 164. Theimage capture times 112 and the pre-determined groups of images andvideos called event clusters 164 are input to the calendar labelannotator 114 along with the personal calendar 146 for producingannotations 118 that are associated with the event clusters 164 in adatabase 120. This alternative embodiment illustrates the utility of thepresent invention for annotating groups of images and videos in additionto the aforementioned utility of annotating single images and videosfrom a collection of digital images and videos 104. In this embodiment,the comparing step performed by the time comparer 156 of FIG. 3 includesassociating pre-determined groups of images with a special event bycomparing the image capture times 112 associated with groups of imageswith the special event times.

A further alternative embodiment is illustrated in FIG. 6. In thisembodiment, the database 120 containing annotations 118 from thecalendar label annotator 114 is queried 122 by a user to find images ofinterest. For example, the user can search for images of interest byentering the query 122 of “mom”. The query 122 is input to a keywordexpander 170 which expands the terms in the query 122 by augmentingadditional related words. The keyword expansion can be performed usingtechniques from the field of natural language expansion. For example,the database WordNet, maintained by Princeton University and availableon-line at http://wordnet.princeton.edu/, can be used to determine theword sense, part of speech, synonyms, hyponyms, hypernyms, etc. Thekeyword expander 170 then outputs an expanded query 172. For example,when the query 122 is “mom” the expanded query 172 is “mom”, “mother”,and “parent”. Each of the additional query words added by the keywordexpander 170 has an associated keyword score based on the strength ofthe relationship between the additional query word and the originalquery 122. For example, “mother” has a keyword score of 1.0 because itis a synonym for “mom” and “parent” has a keyword score of 0.4 becauseit has a related but not equivalent definition. The expanded query 172is used to search the database 120 for images and videos havingannotations related with the expanded query 172 terms. Continuing withthe example, images and videos with the associated annotation “Mother'sDay” would be detected and returned as the query results 124 fordisplaying to the user on the display device 50 such as shown in FIG. 1.Query results 124 can be sorted according to a relevance score, thekeyword score, the capture time of the image or video, alphabeticallyaccording to the name of the image or video, or the like. A user caninspect the results and use manual tools to refine mistakes made by theautomatic retrieval of the images and videos.

In another example, the query 122 for “animals” is processed by thekeyword expander 170 to yield the expanded query “creatures”, “zoo”, and“farm”. When the database 120 is searched for relevant images andvideos, images labeled “Zoo Trip Apr. 6, 2005” are found and returned asthe query results 124.

Annotations 118 stored in the database 120 are based on the specialevent names from the personal calendar 146. Additional annotations canbe generated to store in the database from the special event names.Preferably, additional annotations are hypernyns of the special eventnames. For example, if the special event name is “dog show” thenadditional annotations could be “animal”, “creature” and “living thing”.

A further alternative embodiment is illustrated by FIG. 7. The digitalimage and video collection 104 is input to a person finder 106 forautomatic detection of people in the images and videos. The calendarlabel annotator 114 inputs personal features 110, one set per persondetected by the person finder 106, and also inputs appearance models 116_(1-N) of N different persons of interest. The calendar label annotator114 also inputs image capture times 112 and special events from thepersonal calendar 146. As described hereinabove, the special events cancontain the names (identities) of people present at a special event. Thecalendar label annotator 114 then recognizes persons of interest fromthe persons found by the person finder 106 using the appearance models116, the image capture time 112 and the special events from the personalcalendar 146.

When the special event contains a list of names of people present at aspecial event, then it is more likely than normal that these people willappear in images or videos captured at the event (i.e. images and videosof interest 158 for that particular event). Thus, in recognizing peoplein images and videos of interest for a particular event, greater weight(probability) is placed on people who have both appearance models 116and were present at the special event of interest. This embodimentproduces more accurate recognition of people in a digital image andvideo collection 104.

A further embodiment is shown in FIG. 8. In this embodiment, the list ofspecial events 154 is presented to the user via the display device 50.The user then selects a subset of the displayed special events, therebydefining a list of special events of interest 180. The list of specialevents 154 can be displayed as a text list of the titles of the specialevents and the associated calendar times. Alternatively, the list ofspecial events can be displayed as a calendar with the positions of thetitles of the special events indicative of the calendar times associatedwith the special events. This type of calendar display is commonly knownand practiced for calendar applications. The title of a particularspecial event can be replaced by or augmented by an image or video ofinterest 158 associated with that particular special event. The list ofspecial events of interest 180 is then passed to the time comparer 156for finding images and videos of interest 158 associated with thespecial events of interest 180. This interface allows the user toretrieve images of interest for annotation from only the special eventsof interest 180 rather than all of the special events 154.

A still further embodiment is shown in FIG. 9. The personal calendar 146is selected from among a set of personal calendars 242 by the calendarselector 244. The calendar selector 244 selects the personal calendar146 for passing to the calendar label annotator 114 that corresponds tothe identity of a photographer 232.

The images and videos of the collection 104 are also analyzed by aphotographer determiner 238 to determine the identity of the particularphotographer for each image and video. When the EXIF file format is use,the identity of the photographer 232 can be stored in the “CameraOwner”, “Image Creator”, “Photographer”, or “Copyright” tags forexample. The identity of the photographer of an image or video can beenterer manually before, during, or after capturing the video.Furthermore, several cameras (e.g. in U.S. Pat. Application Publication20020080256A1) have been described that contain means for extractingbiometric information from the photographer 232, identifying thephotographer 232, and then annotating the image with the identity of thephotographer 232. In any case, the photographer determiner 238 discoversthe identity of the photographer 232 and passes that information to thecalendar selector 244. In some cases, the photographer 232 cannot beidentified by the photographer determiner 238. In this case, thephotographer 232 is “unknown”. For example, this situation can occurwhen a person who owns the camera is on vacation and asks a stranger touse her (the vacationing camera owner) camera to capture an image of herin front of a landmark. A camera such as described in U.S. Pat.Application Publication 20020080256A1 can only feasibly identify thephotographer 232 from a small set of potential camera users (e.g. theprimary user is probably the camera owner, and secondary users arefriends and family of the camera owner) whose profiles are known by thecamera. In this case, an image captured by a stranger using the camerawould simply be identified by the photographer determiner 238 as havingan “unknown” photographer 232.

For example, each member of a family can have a personal calendar 146.The selection of the personal calendar 146 from a set of personalcalendars can be based on the identity of the photographer 232 of eachimage or video. For example, an image captured by Jim is input to thecalendar label annotator 114 for comparing the image capture time withthe special event times associated with Jim's calendar.

FIG. 10 is a block diagram of a digital camera phone 300 based imagingsystem that can implement the present invention. The digital cameraphone 300 is one type of image capture device 10. Preferably, thedigital camera phone 300 is a portable battery operated device, smallenough to be easily handheld by a user when capturing and reviewingimages. The digital camera phone 300 produces digital images that arestored using the image data/memory 330, which can be, for example,internal Flash EPROM memory, or a removable memory card. Other types ofdigital image storage media, such as magnetic hard drives, magnetictape, or optical disks, can alternatively be used to provide theimage/data memory 330.

The digital camera phone 300 includes a lens 304 which focuses lightfrom a scene (not shown) onto an image sensor array 314 of a CMOS imagesensor 310. The image sensor array 314 can provide color imageinformation using the well-known Bayer color filter pattern. The imagesensor array 314 is controlled by timing generator 312, which alsocontrols a flash 302 in order to illuminate the scene when the ambientillumination is low. The image sensor array 314 can have, for example,1280 columns×960 rows of pixels.

In some embodiments, the digital camera phone 300 can also store videoclips, by summing multiple pixels of the image sensor array 314 together(e.g. summing pixels of the same color within each 4 column×4 row areaof the image sensor array 314) to create a lower resolution video imageframe. The video image frames are read from the image sensor array 314at regular intervals, for example using a 15 frame per second readoutrate.

The analog output signals from the image sensor array 314 are amplifiedand converted to digital data by the analog-to-digital (A/D) convertercircuit 316 on the CMOS image sensor 310. The digital data is stored ina DRAM buffer memory 318 and subsequently processed by an digitalprocessor 320 controlled by the firmware stored in firmware memory 328,which can be flash EPROM memory. The digital processor 320 includes areal-time clock 324, which keeps the date and time even when the digitalcamera phone 300 and digital processor 320 are in their low power state.

The processed digital image files are stored in the image/data memory330. The image/data memory 330 can also be used to store the user'spersonal calendar information, as will be described later in referenceto FIG. 11. The image/data memory can also store other types of data,such as phone numbers, to-do lists, and the like.

In the still image mode, the digital processor 320 performs colorinterpolation followed by color and tone correction, in order to producerendered sRGB image data. The digital processor 320 can also providevarious image sizes selected by the user. The rendered sRGB image datais then JPEG compressed and stored as a JPEG image file in theimage/data memory 330. The JPEG file uses the so-called “Exif” imageformat described earlier. This format includes an Exif applicationsegment that stores particular image metadata using various TIFF tags.Separate TIFF tags can be used, for example, to store the date and timethe picture was captured, the lens f/number and other camera settings,and to store image captions. In particular, the ImageDescription tag canbe used to store labels, as will be described later in reference to FIG.11. The real-time clock 324 provides a date/time value, which is storedas date/time metadata in each Exif image file.

The digital processor 320 also creates a low-resolution “thumbnail” sizeimage, which can be created as described in commonly-assigned U.S. Pat.No. 5,164,831, entitled “Electronic Still Camera Providing Multi-FormatStorage Of Full And Reduced Resolution Images” to Kuchta, et al., thedisclosure of which is herein incorporated by reference. The thumbnailimage can be stored in RAM memory 322 and supplied to a color display332, which can be, for example, an active matrix LCD or organic lightemitting diode (OLED). After images are captured, they can be quicklyreviewed on the color LCD image display 332 by using the thumbnail imagedata.

The graphical user interface displayed on the color display 332 iscontrolled by user controls 334. The user controls 334 can includededicated push buttons (e.g. a telephone keypad) to dial a phone number,a control to set the mode (e.g. “phone” mode, “calendar” mode” “camera”mode), a joystick controller that includes 4-way control (up, down,left, right) and a push-button center “OK” switch, or the like.

An audio codec 340 connected to the digital processor 320 receives anaudio signal from a microphone 342 and provides an audio signal to aspeaker 344. These components can be used both for telephoneconversations and to record and playback an audio track, along with avideo sequence or still image. The speaker 344 can also be used toinform the user of an incoming phone call. This can be done using astandard ring tone stored in firmware memory 328, or by using a customring-tone downloaded from the mobile phone network 358 and stored in theimage/data memory 330. In addition, a vibration device (not shown) canbe used to provide a silent (e.g. non audible) notification of anincoming phone call.

A dock interface 362 can be used to connect the digital camera phone 300to a dock/charger 364, which is connected to the general controlcomputer 40. The dock interface 362 may conform to, for example, thewell-know USB interface specification. Alternatively, the interfacebetween the digital camera 300 and the image capture device 10 can be awireless interface, such as the well-known Bluetooth wireless interfaceor the well-know 802.11b wireless interface. The dock interface 362 canbe used to download images from the image/data memory 330 to the generalcontrol computer 40. The dock interface 362 can also be used to transfercalendar information from the general control computer 40 to theimage/data memory in the digital camera phone 300. The dock/charger 364can also be used to recharge the batteries (not shown) in the digitalcamera phone 300.

The digital processor 320 is coupled to a wireless modem 350, whichenables the digital camera phone 300 to transmit and receive informationvia an RF channel 352. The wireless modem 350 communicates over a radiofrequency (e.g. wireless) link with a mobile phone network 358, such asa 3GSM network. The mobile phone network 358 communicates with a photoservice provider 372, which can store digital images uploaded from thedigital camera phone 300. These images can be accessed via the Internet370 by other devices, including the general control computer 40. Themobile phone network 358 also connects to a standard telephone network(not shown) in order to provide normal telephone service.

FIG. 11 is a flow chart of yet another embodiment of the presentinvention. This embodiment can use the digital camera phone 300 basedimaging system described earlier in reference to FIG. 10. In thisembodiment, the calendar information is stored in the digital cameraphone 300 and accessed in order to provide appropriate labels fordigital images as they are captured.

In block 400, special events are recorded in the personal calendar 146.This can be done using general control computer 40. As described earlierin reference to FIG. 2, the special events recorded in the personalcalendar 146 can include appointments 140 and occasions 142. Thesespecial events are labels for blocks of time that are personalized tothe user of the digital camera phone 300.

In block 402, the personal calendar 146 is transferred to the digitalcamera phone 300. The transfer can be accomplished by using thedock/recharger 364 and dock interface 362 to transfer the calendar datafrom a hard drive (not shown) of the general control computer 40 to thedigital processor 320 in the digital camera 300. The digital processor320 then stores the personal calendar 140 in the image/data memory 330.

The digital camera phone 300 includes user controls 334 that enable theuser to select various operating modes. In the “phone” mode, the digitalcamera phone 330 operates as a standard mobile phone. In the “calendar”mode, the digital camera phone 330 displays calendar information on thecolor display 332. This enables the user to review and modify theappointments 140 or occasions 142 stored as part of the personalcalendar 146 in the image/data memory 330. In a “camera” mode, thedigital camera phone 300 operates as a still or video camera, in orderto capture, display, and transfer images.

In block 404, the user selects the camera mode. In block 406, the usercomposes the image(s) to be captured, using the color display 332 as aviewfinder, and presses one of the user controls 334 (e.g. a shutterbutton, not shown) to capture the image(s). The image signals providedby the image sensor array 314 are converted to digital signals by A/Dconverter circuit 316 and stored in DRAM buffer memory 318.

In block 408, the digital processor 320 reads the value of the real timeclock 324 after each image is captured, to determine the date and timethe picture was taken. In block 412, the digital processor 320 retrievesthe calendar entry for the date/time provided by the real time clock324.

In block 414, the digital processor 320 determines if the calendar entryfor the current date/time corresponds to a special event time, as wasdescribed earlier in reference to FIG. 3.

In block 416, the digital processor 320 uses the calendar entry tocreate proposed image metadata. For example, if the calendar entry is“Matthew's Soccer game”, the proposed image metadata could be “Event:Soccer game, Subject: Matthew”.

In block 418, the proposed image metadata is displayed on the colordisplay 332 along with the image(s) captured in block 406. This enablesthe user to see the proposed image metadata, and check whether or not itis an appropriate label for the captured image(s). In addition todisplaying the proposed image metadata, the processor 320 displays arequest for the user to approve the proposed metadata. This can be done,for example, by displaying the text “OK?” along with “yes” and “no”selectable responses, on the color display 332.

In block 420, the user selects either the “yes” or “no” response, usingthe user controls 334. If the metadata is not appropriate, the userselects the “no” response. This can happen if the special event did nottake place, for example if the soccer game was cancelled, so that theimages correspond to a different type of event.

In block 422, if the user does not “OK” the metadata, (“no” to block420) the digital processor 320 displays a user interface screen on thecolor display 332 which enables the user to modify the metadata. Thiscan be done by simply deleting the metadata, or by selecting alternatemetadata. The alternate metadata can be selected from a list offrequently used labels (e.g. Science museum, playground) or can bemanually entered text strings.

It should be noted that once the user has approved or modified themetadata for one image (using blocks 412-422) in order to create anacceptable label for the images, there is normally no need to repeatthese steps during the same photo event. Thus, the approved (ormodified) metadata can be automatically stored in the image files of allsubsequent images taken of the same photo event (for example for allimages taken until the camera is turned off), without repeating blocks416-422.

In block 424, if the user “OKs” the metadata, (“yes” to block 420), orif the user provides modified metadata in block 422, the digitalprocessor 320 stores the metadata in the image file(s) of the capturedimage(s). For example, the metadata can be stored in theImageDescription tag of the Exif file which contains the captured stillimage.

In block 426, the image files are transferred to the database 120. Thiscan be done, for example, by using the wireless modem 350 to transmitthe image files over the mobile phone network 358 to the photo serviceprovider 372. The photo service provider 372 can then store the imagefiles, and enable them to be accessed by various computers, includinggeneral control computer 40, over the Internet 370. The image files canalso be accessed by the digital camera phone 300, using the mobile phonenetwork 358. Alternatively, the image files can be transferred to thegeneral control computer 40 using the dock interface 362 anddock/recharger 364. The metadata in the image file, such as theDate/Time metadata and the special event labels stored using theImageDescription tag, can also be read from each image file and storedin a separate metadata database along with the image name, to enablemore rapid searching.

In block 430, the metadata of the database 120 is searched to locateimages of interest. This can be accomplished by entering the query 122,as described earlier in reference to FIG. 6.

In block 432, the images having metadata which best match the query 122are displayed. If the images are stored in the general control computer40, they can be displayed on the display device 50. Alternatively, ifthe images are stored by the photo service provider 372, they can betransferred to the digital camera phone 300 using the mobile phonenetwork 358 and displayed on the color display 332.

In block 434, the user can modify the metadata associated withparticular photo events, in order to correct or augment the metadatalabels. The modified metadata labels are then stored in the database120.

FIG. 12 is a flow chart of a still further embodiment of the presentinvention. In this embodiment, different special events are associatedwith different list of individuals (such as family, friends, team mates,work associates, etc.) who the user would like to be able to access theimages of that type of special event taken by the user. For example, theuser may want to allow all of Sarah's relatives to view images ofSarah's birthday, and may want to allow all of Matthew's team mates toview images of Matthew's soccer game.

In block 401, special events are recorded in a personal calendar 146, asdescribed earlier in reference to block 400 of FIG. 11. In addition, thepersonal calendar 146 is made accessible over a network. This can bedone by enabling the photo service provider 372 to access the personalcalendar 146 via the Internet 370 .

In block 403, share lists for at least some of the special eventsrecorded in the personal calendar 146 are stored. This can be done, forexample, by enabling the user to select, from the user's share listalready provided by the photo service provider 372, those users thatwill be allowed to access images associated with different types ofspecial events. For example, there might be a first list of family andfriends that are allowed to access holiday and birthday images, a secondlist of teammates that are allowed to access images of team events, anda third list of work colleagues that are allowed to access images ofwork associated events. In addition, an event-specific share list can becreated automatically by adding all of the participants of a particularevent recorded in a personal calendar (e.g. all of the people invited toa party or a meeting) to the share list for that special event.

In block 404, the user selects the camera mode and in block 406, theuser composes the image(s) to be captured, as was described earlier inreference to FIG. 11.

In block 408, the digital processor 320 reads the value of the real timeclock 324 after each image is captured, to determine the date and timethe picture was taken. In block 409, the date/time is stored as metadatais association with the captured image(s), for example in the Date/TimeTIFF tag of the Exif image file.

In block 411, the digital processor 320 in the digital camera phone 300initiates the transfer of the captured image(s) to the database. Thiscan be done automatically after each image is captured or after acertain number of images are captured, or can be manually initiated bythe user. Once initiated, the wireless modem 354 begins to transmit theimage file(s) over the mobile phone network 358 to the photo serviceprovider 372.

In block 412, photo service provider retrieves the calendar entry forthe date/time stored in the transferred image file.

In block 414, the service provider 372 determines if the calendar entryfor the current date/time corresponds to a special event time, as wasdescribed earlier in reference to FIG. 11.

In block 417, the service provider 372 uses the photo event to createimage metadata and a proposed “share list”, which is one of the sharelists stored in block 403. For example, if the calendar entry is“Matthew's Soccer game”, the image metadata could be “Event: Soccergame, Subject: Matthew”, and the proposed share list could be the listof all of Matthew's teammates.

In block 419, the share list is communicated to the digital camera phone300 via the wireless modem 350, and is displayed on the color display332 along with the image(s) captured in block 406. This enables the userto decide whether or not to share the captured images with the sharelist associated with the identified special event. In addition todisplaying the share list (which can list individuals or the name of thegroup), the processor 320 displays a request for the user to approve theproposed share list. This can be done, for example, by displaying thetext “OK?” along with “yes” and “no” selectable responses, on the colordisplay 332.

In block 421, the user selects either the “yes” or “no” response, usingthe user controls 334. If the share list is not appropriate, the userselects the “no” response. This can happen if the special event did nottake place, for example if the soccer game was cancelled, so that theimages correspond to a different type of event.

In block 423, if the user does not “OK” the share list, (“no” to block421) the digital processor 320 displays a user interface screen on thecolor display 332 which enables the user to modify the share list. Thiscan be done by simply not allowing the images to be shared with anyoneelse, or by selecting an alternative share list.

It should be noted that once the user has approved or modified the sharelist for one image (using blocks 412-423), there is normally no need torepeat these steps during the same photo event. Thus, the approved (ormodified) share list can be automatically used for all subsequent imagestaken of the same photo event (for example for all images taken untilthe camera is turned off), without repeating blocks 414-423.

In block 425, if the user “OKs” the share list, (“yes” to block 421), orif the user provides a modified share list in block 423, the serviceprovider 372 enables the uploaded images associated with the specialevent to be shared with those uses on the share list. The images can beshared using methods well-known in the prior art, for example by sendingan email to each individual on the share list which contains a link toenable the images to be view using a web browser. The individuals on anyshare list can also be given other types of authorization, for exampleto order prints of the transferred images, as described in commonlyassigned U.S. Pat. No. 5,760,917 to Sheridan, the disclosure of which isherein incorporated by reference.

In block 430, the metadata of the database 120 is searched to locateimages of interest. The search can be performed either by the user ofthe digital camera 300, who has access to all of the images, or toindividuals on one of the user's share lists, who have access to onlythose images associated with particular special events. This search canbe accomplished by entering the query 122, as described earlier inreference to FIG. 6.

In block 432, the images having metadata which best match the query 122are displayed. The images can be displayed on the display device 50 ofthe general control computer 40 or they can be transferred to thedigital camera phone 300 using the mobile phone network 358 anddisplayed on the color display 332.

FIG. 13 is a flow chart of an additional embodiment of the presentinvention. In this embodiment, particular special events (such asimportant meetings, concerts, weddings, etc.) are recorded as silentevents in the personal calendar. In addition, ring tones can beassociated with other special events, such as an short excerpt from“Stars and Stripes forever” for the 4^(th) of July holiday, or “Happybirthday” for a birthday anniversary. When an incoming phone call isreceived by the user's phone camera, the personal calendar is checked todetermine if the current date/time corresponds to a silent event. If itdoes, the vibration device is used to indicate to the user that there isan incoming call, instead of using a ring tone. If not, the personalcalendar is checked to determine if a special ring tone should be used,instead of the default ring tone, to indicate the incoming call.

In block 400, special events are recorded in the personal calendar 146.This can be done using general control computer 40. As described earlierin reference to FIG. 2, the special events recorded in the personalcalendar 146 can include appointments 140 and occasions 142. Thesespecial events are labels for blocks of time that are personalized tothe user of the digital camera phone 300.

In block 436, the user can record whether some special events are silentevents, during which they do not want their phone to ring. These silentevents may include particular work-related meetings or social events,such as weddings, concerts, etc., where a ringing phone would annoyothers and embarrass the user. The user can also specify that somerecurring events (such as staff meetings or concerts) should always besilent events, so that they are automatically recorded as silent eventswhether a new event of this type is added to their calendar in block400.

In block 438, the user can assign specific ring tones to specificspecial events. For example, the user can assign holiday ring tones(e.g. a Christmas theme, Thanksgiving theme, Halloween theme) to theseholiday special events, a “Happy Birthday” song ring tone to birthdayanniversary special events, etc.

In block 440, the personal calendar 146 and the assigned ring tones aretransferred from the general control computer 40 to the digital cameraphone 300, and stored in the image/data memory 330.

In block 442, the digital processor 320 detects an incoming phone callfrom the mobile phone network 358 via the wireless modem 350.

In block 408, the digital processor 320 reads the value of the real timeclock 324 to determine the current date and time. In block 412, thedigital processor 320 retrieves the personal calendar entry for thedate/time provided by the real time clock 324.

In block 444, the digital processor 320 determines if the calendar entryfor the current date/time corresponds to a silent event recorded inblock 436.

In block 446, if the current date/time corresponds to a silent event(yes to block 444), the vibration device described earlier in referenceto FIG. 10 is used to indicate that there is an incoming call.

In block 448, if the current date/time does not correspond to a silentevent (no to block 444), the digital processor 320 determines if thecurrent date/time corresponds to a special event which has been assigneda special ring tone.

In block 450, the digital processor 320 in the digital camera phone 300uses either the special ring tone determined in block 448, or thedefault ring tone, to indicate to the user that there is an incomingphone call.

It should be noted that in some embodiments, different ring tones can beassigned to different callers. In this case, a particular special ringtone may be used only for certain callers. For example, the “Happybirthday” ring tone can be used only for calls from a phone numberassociated with the person who's birthday anniversary is the specialevent in the user's personal calendar. This can remind the user toconvey a “happy birthday” message to the caller.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   10 image capture device-   20 microprocessor unit (aka digital image processor)-   30 image output device-   40 general control computer-   50 display device-   60 keyboard-   104 digital image and video (also referred to as “collection”)-   106 person finder-   108 capture time extractor-   110 personal features-   112 image capture time-   114 calendar label annotator-   116 appearance models-   118 annotations-   120 database-   122 queried or query-   124 query results-   140 appointment-   142 occasion-   144 journal entry-   146 personal calendar-   150 user input-   152 event processor-   154 special event list-   156 time comparer-   158 digital images and videos of interest-   162 event clusterer-   164 event clusters-   170 keyword expander-   172 expanded query-   180 events of interest-   232 photographer-   238 photographer determiner-   242 personal calendars-   244 calendar selector-   300 digital camera phone-   302 flash-   304 lens-   310 CMOS image sensor-   312 timing generator-   314 image sensor array-   316 A/D converter circuit-   318 DRAM buffer memory-   320 digital processor-   322 RAM memory-   324 real-time clock-   328 firmware memory-   330 image/data memory-   332 color display-   334 user controls-   340 audio codec-   342 microphone-   344 speaker-   350 wireless modem-   352 RF channel-   358 phone network-   362 dock interface-   364 dock/charger-   370 Interent-   372 service provider-   400 block-   401 bock-   402 block-   403 block-   404 block-   404 block-   406 block-   406 block-   408 block-   408 block-   409 block-   411 block-   412 block-   414 block-   414 block-   416 block-   417 block-   418 block-   419 block-   420 block-   421 block-   422 block-   423 block-   424 block-   425 block-   426 block-   430 block-   432 block-   434 block-   436 block-   438 block-   440 block-   442 block-   444 block-   446 block-   448 block-   450 block

1. A method for associating event times or time periods with digitalimages in a collection for determining if a digital image is ofinterest, comprising: a) storing a collection of digital images eachhaving an associated capture time; b) comparing the associated capturetime in the collection with a special event time to determine if adigital image in the collection is of interest, wherein the comparingstep includes calculation of a special event time associated with aspecial event based on the calendar time associated with the specialevent and using such information to perform the comparison step; and c)associating digital images of interest with the special event.
 2. Themethod of claim 1, further including: d) labeling a captured digitalimage with special event identifier information.
 3. The method of claim1, wherein the comparing step further includes collecting a list ofspecial events and related special event time for each collected specialevent, respectively.
 4. The method of claim 1, wherein the associatingstep includes storing in a database digital images of interest with aspecial event.
 5. The method of claim 1, wherein the comparing stepincludes user entry of a special event time associated with a specialevent and using such information to perform the comparison step.
 6. Themethod of claim 1, wherein the comparing step includes associatingpre-determined groups of images with a special event.
 7. The method ofclaim 1, wherein a special event is identified from a journal entry, anappointment, or an occasion or a combination thereof.
 8. The method ofclaim 1, wherein a special event is identified by user inspection orautomatically or both.
 9. The method of claim 1, further including: d)supplying a query; e) retrieving images and videos using the specialevent and the query; and f) displaying the retrieved images and videoson a display.
 10. A method for associating event times or time periodswith digital images in a collection for determining if a digital imageis of interest, comprising: a) storing a collection of digital images oreach having an associated capture time; b) clustering chronologicallyrelated digital images into one or more event clusters; c) comparing theassociated capture times of images in the event clusters with a specialevent time to determine if an event cluster is of interest; and d)associating an event cluster of interest with the special event.
 11. Amethod for associating event times or time periods with digital imagesin a collection for determining if a digital image is of interest,comprising: a) storing a collection of digital images each having anassociated capture time; b) providing a list of events with associatedevent times from a personal calendar; c) classifying an event as aspecial event; d) comparing the associated capture times in thecollection with an event time associated with the special event todetermine if a digital image in the collection is of interest; and e)associating digital images and videos of interest with the specialevent.
 12. The method of claim 11, wherein the classification of anevent as a special event includes determining a likelihood that imageswill be captured at the event.
 13. The method of claim 11, wherein theclassification of an event as a special event includes using user inputindicating whether images will be captured at the event.
 14. A methodfor associating event times or time periods with digital images in acollection for determining if a digital image is of interest,comprising: a) storing a collection of digital images each having anassociated capture time and an associated photographer; b) providing apersonal calendar of special events with the associated photographer c)comparing the associated capture times in the collection with a specialevent time to determine if a digital image in the collection is ofinterest; and d) associating digital images of interest with the specialevent.
 15. The method of claim 14, further including: e) labelingdigital images of interest with special event identifier informationassociated with the special event.
 16. The method of claim 14, whereinthe comparing step further includes collecting a list of special eventsand related special event time for each collected special event,respectively.
 17. The method of claim 14, wherein the associating stepincludes storing in a database digital images of interest with a specialevent.
 18. The method of claim 14, wherein the comparing step includesuser entry of a special event time associated with a special event andusing such information to perform the comparison step.
 19. The method ofclaim 14, wherein the comparing step includes calculation of a specialevent time associated with a special event based on the calendar timeassociated with the special event and using such information to performthe comparison step.
 20. A method for labeling digital images capturedusing a digital capture device, comprising: a) receiving and storingpersonal calendar information in a digital capture device, the personalcalendar information including occasion or appointment information; b)capturing and storing a digital image in the digital capture device; c)determining a capture time for the captured digital image; d)automatically comparing the capture time with the personal calendarinformation to determine a special event label; and e) storing thespecial event label in association with the captured digital image. 21.The method of claim 20, further including displaying the special eventlabel on a display of the digital capture device.
 22. The method ofclaim 21, further including providing a user interface screen on thedisplay to enable a user to approve the special event label.
 23. Themethod of claim 21, further including transferring the captured digitalimage and the special event label over a wireless network.
 24. A methodfor enabling a plurality of different users to access particular digitalimages in a collection of digital images, comprising: a) storing datalisting a plurality of special events and the subset of the plurality ofdifferent users associated with each of the special events; b) storing acollection of digital images each having an associated capture time; c)determining a particular special event for at least one associatedcapture time; and d) enabling the users associated with the particularspecial event to access the digital image having the associated capturetime.
 25. The method of claim 24, further including displaying a list ofthe users associated with the particular special event on a display. 26.The method of claim 25, further including providing a user interfacescreen on the display to enable the list of users to be approved.
 27. Amethod for indicating an incoming phone call in a mobile phone,comprising: a) storing personal calendar information in the mobilephone, the personal calendar information including occasion orappointment information and information indicating that at least oneoccasion or appointment is a silent event; b) detecting an incomingphone call from a mobile phone network; c) determining a current dateand time; d) comparing the current date and time with the personalcalendar information to determine whether the current date and timecorresponds to the silent event; e) if the current date and timecorresponds to the silent event, providing a non-audible indication ofthe incoming call; and f) if the current data and time does notcorrespond to the silent event, providing an audible indication of theincoming call.
 28. The method of claim 27 further including: g) storing,in the personal calendar, information associating a custom ring tonewith one of the occasions or appointments; h) comparing the current dateand time with the personal calendar information to determine whether thecurrent date and time corresponds to the occasion or appoint associatedwith the custom ring tone; and h) if the current data and timecorresponds to the occasion or appointment associated with the customring tone, using the custom ring tone to provide the audible indicationof the incoming call.